DSC 478

Jonathan Lynch

Assignment 2 - Part 3: Data Analysis & Predictive Modeling on Census Data

3-A: Preprocessing and data analysis

Import libraries and read in file:

Examine data for missing values, replace missing values with the attribute mean for numeric variables, & remove instances with missing values for categorical attributes:

Replace all question marks with null, and see which attributes contain missing values:

Take a look at the data types:

Convert age variable to float (so can replace missing values with the mean):

Replace missing values with the attribute mean for age variable, and then remove rows containing missing values:

Round off age at decimal point, and convert to an integer:

Double-check all missing values were removed:

Examine the characteristics of the attributes, including relevant statistics for each attribute, histograms showing distributions for numeric attributes, & bar graphs showing value counts for categorical attributes:

Look at the descriptive statistics of the numerical variables for dataset:

The above descriptive statistics of the numeric attributes of this dataset shows the mean, median, and standard deviation for age, education and hours-per-week. The minimum and maximum age, education and hours-per-week is also shown, along with the first quartile (Q1/25th percentile) and third quartile (Q3/75th percentile). The total count is also shown.

The mean age is 38.4 (years), the median age is 37 (years), and the standard deviation for age is 12.962 (years). The minimum age is 17 (years) and the maximum age is 90 (years). The first quartile (Q1/25th percentile) for age is 28 (years) and the third quartile (Q3/75th percentile) for age is 47 (years).

The mean education is 10.125 (years), the median education is 10 (years), and the standard deviation for education is 2.542 (years). The minimum education is 1 (years) and the maximum education is 16 (years). The first quartile (Q1/25th percentile) for education is 9 (years) and the third quartile (Q3/75th percentile) for education is 13 (years).

The mean hours-per-week is 41.08 (hours), the median hours-per-week is 40 (hours), and the standard deviation for hours-per-week is 11.88 (hours). The minimum hours-per-week is 1 (hour) and the maximum hours-per-week is 99 (hours). The first quartile (Q1/25th percentile) for hours-per-week is 40 (hours) and the third quartile (Q3/75th percentile) for hours-per-week is 45 (hours).

Show the distributions of the numeric variables through histograms:

Looking at the histogram for the numeric age variable, it appears to have a positively skewed distribution.

Looking at the histograms for the education and hours-per-week numeric variables, the distribution of the education attribute appears to be slightly negatively skewed (with a large number of observations consolidated primarily within 3 distinct years), and the histogram of hours-per-week shows that 40 hours per week seems to be by far the most popular number of hours worked per week (substantially outweighing all other values).

Show value counts for categorical variables through bar graphs:

Looking at the above bar graphs for the workclass and marital-status categorical variables, it appears that the vast majority of adults are employed in the private sector (nearly 7,000), and far fewer work in the public sector (~ 1,250) or are self-employed (~ 1,100). The bar plot of the marital-status attritube shows that nearly half of the adults are married (~ 4,700), and half are single (~ 4,700).

Looking at the above bar graphs of the categorical variables race and sex, it appears that the vast majority of adults in this dataset are white (~ 8,000), and less are black (~ 800), Asian (~ 250), American Indian (~ 100), or Hispanic (~ 100). The bar plot of the sex variable shows that the majority of adults in this dataset are male (~ 6,300), and fewer are female (~ 3,000).

The above bar graph of the categorical income variable shows that the majority of adults (~ 7,000) in this dataset have an income of less than or equal to 50,000, and fewer (~ 2,250) have an income of greater than 50,000.

Perform cross-tabulations of education+race, work-class+income, work-class+race, and race+income (including bar graphs), create a table showing percentages of each race category that fall in the low-income group, and discuss results:

Cross-tabulation of education+race:

From the above analyses, it is evident that the mode educational level (number of years) achieved across all races except for the Asian category of race is 9 (years). The mode education for the Asian race is 13 (years). Also, there appears to be three distinct education levels (9 years, 10 years, & 13 years) that have by far the greatest number of adults falling into.

Cross-tabulation of work-class+income:

From the cross-tabulation between work-class & income, we can see that the private sector employs both the most low income (<= 50K) and high income (> 50K) adults, compared with the public sector and those that are self-employed. This is to be expected, however, as the private sector overall employs a considerably larger proportion of adults compared with both the public sector and those that are self-employed.

Cross-tabulation of work-class+race:

From the cross-tabulation between work-class and race, can see that the private sector category of the work-class attrtibute employs by far the most adults across all race categories. Again, this is to be expected due to the fact that the private sector employs bar far the greatest proportion of adults.

Cross-tabulation of race+income:

The cross-tabulation of race & income shows that whites are the highest earning race category (with the highest amount of both low-income adults and high-income adults).

Table of percentages of each race that fall in the low income group (<=50K):

The above table shows the percentages of each race that fall into the low income group (<= 50K). Whites have the lowest percentage (73.7%) falling into the low income category (<= 50K). Consequently, whites will implicity have the highest percentage of adults which fall into the high income category (not shown).

Discuss observations from analyses:

From the above analyses, it is evident that the mode educational level (number of years) achieved across all races except for the Asian category of race is 9 (years). The mode education for the Asian race is 13 (years). From the cross-tabulation between work-class & income, we can see that the private sector employs both the most low income (<= 50K) and high income (> 50K) adults, compared with the public sector and those that are self-employed. We can also see that the private sector category of the work-class attrtibute employs by far the most adults across all race categories from the cross-tabulation between work-class & race. Finally, the cross tabulation of race & income shows that whites are the highest earning race category. This is also evident from analyzing the above table of percentages of each race that fall into the low income group (<= 50K). Whites have the lowest percentage (73.7%) falling into the low income category (<= 50K). Consequently, whites will implicity have the highest percentage of adults which fall into the high income category (not shown).

Compare and contrast the characteristics of the low-income and high-income categories across the different attributes:

Compare income & age:

Looking at how the low-income (<= 50K) and high-income (> 50K) categories compare with the age attribute, we can see that younger ages tend to fall predominately in the low-income category. As age increases, there is an increase in the number of adults that fall into the high-income category (particularly from the early 30's to the mid to late 50's). However, the number of adults falling into the high-income category never exceeds the number of adults falling into the low-income category at any age. The age of 38 appears to have the highest number of both low-income adults (~ 270) and high-income adults (~ 140).

Compare income & work-class:

Looking at how the low-income and high-income categories compare with work-class, it appears that being self-employed yielded the highest percentage of adults with incomes falling in the high-income class (36.8%). The private sector had the lowest percentage of adults falling in the high-income category (21.6%), and the greatest percentage of adults falling in the low-income category (78.4%). The public category of work-class fell somewhere in the middle.

Compare income & education:

Looking at the above bar graph of income categories by educational level, we can see that the only educational levels where the number of high-income adults exceeded the number of low-income adults was for 14 or more years of education. For the low-income category of adults, the most prevalent number of years of education was 9 years (with over 2,500 adults). For the high-income category of adults, the most prevalent number of years of education was 13 years (with around 700 adults).

Additionally, the above table shows the percentages of low-income and high-income categories for each educational level. One of the most interesting takeaways is that as the years of education increases, the percentage of low-income adults steadily decreases and the percentage of high-income adults steadily increases, indicating that there seems to be a strong correlation between the number of years of education attained and higher incomes.

Compare income & marital status:

Interpreting the above table, it appears that the vast majority of single adults (93.3%) fall into the category of low-income earners, while married adults are more evenly split, with 57.6% of married adults falling into the low-income category and 42.4% of married adults falling into the high-income category.

Compare income & race:

The highest percentage of adults that fall into the high-income category are white (26.27%), followed closely by Asians (~ 23%). Hispanics have the lowest percentage of adults that fall into the high-income category (8%), or the highest percentage falling into the low-income category. Blacks and American Indians also appear to have very high percentages of adults falling into the low-income category (86.7% & 90.2% respectively).

Compare income & sex:

A higher percentage of males (30.7%) fall into the high-income category, compared to that of females (only 11.89%). Conversely, a higher percentage of females (88.11%) fall into the low-income category compared with that of males (69.3%).

Compare income & hours per week:

The highest number of hours worked per week for both low-income adults and high-income adults is 40 hours per week (almost 3,500 low-income adults & nearly 1,000 high-income adults). There also appears to be a spike in hours worked per week at 50 hours (close to 500 low-income adults and around 400 high-income adults).

3-B: Predictive Modeling and Model Evaluation:

Create dummy variables for categorical attributes, separate target attribute (income > 50K), and drop low-income:

Build Naive Bayes (Gaussian) classifier, perform 10-fold cross-validation, & show overall accuracy:

Build decision tree classifier (using "entropy" as selection criteria), perform 10-fold cross-validation, & show overall accuracy:

Build LDA classifier, perform 10-fold cross-validation, & show overall accuracy:

Generate a visualization of decision tree model:

The above decision tree is very difficult to see due to its size. I will submit a separate png file along with this assignment that can be zoomed in on if needed.